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ABSTRACT 



This symposium on assessment and evaluation modeling 
consists of three presentations. "Training Assessment Among Kenyan 
Smallholder Entrepreneurs" (George G. Shibanda, Jemymah Ingado, Bernard 
Nassiuma) reports a study that assessed the extent to which the need for 
knowledge, information, and skills among small scale farmers can promote 
effective management of the off-farm enterprises. It recommends business 
counseling as an integral part of extension education. "Business-Focused 
Evaluation: A New Collaborative Model" (Valerie Stead) presents a case study 
of a business -focused evaluation model developed as part of an organizational 
development intervention for a global semi-conductor company. It assesses the 
model's value against three principles concerning collaboration, joint 
ownership, and integration drawn from client and provider needs and current 
thinking in evaluation research. "Challenges in Measuring and Evaluating 
Diversity Efforts in Human Resource Development" (T. Marshall Egan) examines 
challenges faced by practitioners and scholars interested in evaluation and 
measurement of diversity efforts in organizational contexts. It considers an 
organizational systems model illustrating the diversity-related interchanges 
between an organization and the macro environment and examines challenges to 
measurement and evaluation due to the latency of the diversity, performance, 
and performance improvement constructs. All three papers include substantial 
bibliographies. (YLB) 
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The smallholder occupation includes on-farm and non-farm activities. The latter are by extension 
enterprises supplementing the small farm incomes due to population pressure and reduced 
productive farm holdings in Kenya, and are to raise smallholder farm earnings. One crucial factor 
is lack of rapid development for holders being less responsive to farm innovations, notably 
inadequacies in knowledge and skills. This work recommends that business counseling approach 
need be integral part of extension education for small farm enterprise management. 

Keywords: Needs assessment, Entrepreneur, Management Skills 

The term smallholder is interchangeably used with relative terms like small farmer and peasant in the African 
setting. Leonard (1977) in his research on reaching the peasant farmer describes small farmer and peasant as 
managers of their small individual holding of land for small scale agricultural production and off-farm services. One 
handicap he identifies is lack of rapid small farm development calling for relevant and timely dissemination of 
knowledge and instruction on small farm centered enterprises. According to Livingston (1981) non-farm activities 
simply supplement on-farm earnings. One major reason for the search for non-farm activities on the farm or away 
from the farm is in response to land shortage in areas of population pressure while in others it can as well be an 
indication of good income opportunities. Gundu (1985) study on agricultural information diffusion to smallholders 
in Kenya observed that the paid rural employment opportunities and rural entrepreneurial tasks off-farm are both 
male and female domain. As holders they combine both on-farm and off-farm activities. The varying educational 
level, off-farm enterprises and gender are indicators towards training identification and needs among smallholders 
Odeng(1989). Smallholder entrepreneurs have not been able to realize their full potential due to inadequate training 
in skills relevant to areas of off-farm occupation. Enterprenuers operate to support the smallholder farm incomes. 
Off-farm enterprises fail due to mismanagement and inappropriate knowledge and skills application. Farm 
enterprises have erratic operational tendencies just because small farmers need instant additional income. The 
success of off-farm enterprises is depended on having the required skills. These skills can be described simply as 
technical and managerial. It is therefore being argued here that the attainment of the right skills is achieved through 
proper training followed by the practice of the learnt skills. This research is aimed at identifying the training needs 
in order to improve human resource to meet the day today challenges of running and optimizing financially based 
off-farm activities. Calling for small farmer training presupposes provision of knowledge, information and allied 
skills to ensure proper management of off-farm enterprises. 

Problem Statement 

The hypothesis behind this work is that smallholders are potentially intelligent, interested in obtaining relevant 
knowledge, information and skills with keen desire to utilize them for their personal welfare and provide goods and 
services to themselves as a rural community. This study is therefore aimed at assessing the extend to which the need 
for knowledge, information and skills among small scale farmers can promote effective management of the off-farm 
enterprises. It is argued that the Kenyan smallholder entrepreneur ineffectiveness is due to inappropriate training 
packages, non identification of proper training needs and inadequacies in trainer capacities. It is apparent William 
(1980), Odeng (1989), ILO (1992) that training is a prerequisite for knowledge, information and skills transmission. 
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necessary for achieving better management of the off-farm activities among the smallholders. The major bottleneck 
is Obura (1993) compounded by mismanagement of enterprises as may be revealed through lack of the necessary 
knowledge, information and requisite skills in running off-farm enterprises. Smallholders need to cultivate positive 
attitude towards training and have their training requirements spelt out before any business management training is 
imparted. 

Theoretical Framework 

This study applied a qualitative research data collection approach constituting mainly responses drawn from 
smallholders. The target group studied comprised of entrepreneurs. The focus in on rural development through 
extension education as means of informing, advising and training the smallholder on farm and off-farm activities. 
While focusing on rural development that constitute over 85 percent of smallholder activities, Supe (1983) 
concurred that rural development is the result of many interacting forces where education is one of them. His 
description of educational needs for rural development appears in four groups given as: 

• General or basic education designed for providing literacy and primary education. 

• Family improvement education designed primarily to impart knowledge, skills and attitudes useful in 

improving the quality of family life. 

• Community improvement education designed to strengthen local and national institutions and processes 

through instructions on matters such as local and national government, community projects, etc. 

• Occupational education designed to develop particular knowledge and skills associated with various 

economic activities and useful in making a living. 

. Supe (1983) identifies extension education as a method of education for rural based households outside the 
regularly organized schools for bringing out social and cultural development. Extension in this context is simply 
described as means to extend, to spread or disseminate useful information and ideas to rural people outside the 
regular organized schools and classrooms whilst education is the production of desirable changes in human 
behavior. Extension education is seen therefore as a program for people based on their needs and problems. It is 
basically designed to meet these needs and solve problems. It is a teaching and learning process striving to induce 
the following changes: 

• Changes in knowledge or things known. 

• Changes in skills or things done. 

• Changes in attitudes or things felt. 

Daham and Bhatnagar (1985) in their work on education and communication for development observed 
training to mean to educate a person so as to be fitted, qualified, proficiently in doing some job. Training includes 
education which aims at bringing a desirable change in the behavior of the trainee. This change, they assert that 
requires a change in knowledge, skills, attitudes, values, beliefs and understandings so that the trainee becomes 
qualified and proficient in communicating and performing certain jobs. Training through extension involves not only 
educating the rural people in determining their problems and methods of solving them but also inspiring them 
toward positive action. 

The sessional paper No. 2 of 1992 on small Enterprises and Jua kali Development in Kenya, provides a 
policy framework for promoting small scale and Jua kali enterprise development in Kenya. The major policy 
framework set out by this paper includes: 

(i) Providing direct assistance for management and entrepreneurial training and skills upgrading for 
individual entrepreneurs. 

(ii) Disseminating information on market and appropriate technology 

Jones (1998) summarizes micro enterprises access to training in Zambia as follows: 26% have had access to 
technical training (4 out of 10 have received their training through apprenticeships). Only 17% of women 
entrepreneurs compared with 38% of men are technically trained. 9.2% have received some business training (12% 
for men 6% for women). The surveys in both Kenya and Zambia also found similar positive relationships between 
training and enterprise profitability. The relationship between education and access to credit and the size of the 
enterprise, conducted by Oketch and Kisundu (1995) study on the demand and supply of small enterprises finance in 
Kenya done for the British Overseas Development Agency, found that there is a positive correlation between 




38-1 



education of the owner and the size of the enterprise on the one hand and access to credit and non-financial services 
on the other. The Central Bank (1999), Yambo (1992), Williams (1980), ILO (1992) consider the constraint to 
smallholder entrepreneurs to be counseling of the entrepreneurs. Negative perceptions of the smallholder 
entrepreneurs on the basis of lack of tangible financial trace record, cases of illiteracy are thought to be the worst in 
credit management. 

Research Questions 

The research questions guiding this study are as follows: 

• What is the distribution of holder enterprises on the basis of education levels? 

• To what extend does educational levels show enterprise needs? 

• What attitudes do smallholder need for enterprise operation? 

• What knowledge and skill deficiency exist among smallholder entrepreneurs? 

• What are the objectives of smallholder enterprises? 

• What tasks do smallholders need to perform to actualize entrepreneurial goals? 

• What is the right time for training? 

• What training capacities exist for imparting knowledge and skills? 

Methodology 

The Study Area 

The sample area selected for this study is Kakamega district. The district has very high smallholder 
population density and poverty in Kenya. Smallholder land ownership is between 0.1 to 11 acres on average 
household. Due to pressure of land and reduced land for cultivation, most people have opted for off-farm activities 
to supplement the on farm cultivation just to raise extra incomes to support the families. The district forms part of 
western province whose educational attainment of the small farm population not currently in full-time education 
indicates, Kenya Government (1981) 30.1% male and 39.6 % female had received no education. The 50 + years age 
group showed a no education picture of 37.4 % male and 46.6 % women. 

Target Population ■ 

The target population for the study comprised of 439 smallholders registered as business operators at the 
district trade office. They form part of what Gundu (1986), Shibanda (1999) see as the rural majority but less 
informed and less educated requiring access to agricultural education, information, advice and assistance in order to 
improve their methods of production, marketing, service provision and management of off-farm enterprises. They 
are smallholders whose holdings do not exceeding eleven acres. The study assumes, Kenya Government (1981) 
definition of holder to mean a person or persons who have the control and ability to make decisions relevant to the 
agricultural activities on the holding. A holding means all the land and livestock used partly or completely for 
agricultural purposes where again a holder may in fact operate neither land nor livestock. The purpose of such 
holdings is to provide food purely for subsistence yet cash oriented activities are undertaken to supplement on farm 
earnings. 

Data Collection, Instruments, and Procedures 

The research techniques applied in aid of data collection include: 

• List sampling by drawing samples in list sampling of the District Trade Development Office. This method 

allowed to select a random group of 439 people from trade office on details of registered business 
owners. The details given were mechanical since the names sampled were randomly picked up. The 
persons on the trade register had every equal chance of selection out of which 23 people responded. This 
constituted 5.2 % of the total population. 

• Attitude scales showing how people feel towards certain activities. 

• Knowledge and comprehension test to find out whether the person understand or can apply certain acquired 

knowledge in a given situation. 

• Skill or performance rating used to determine the amount of skill attainment in doing certain activity. 
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The first stage of the study deals with the exploration of relevant data available from documents. Basically 
this constitute information contained in official government records available for scrutiny. Data gathered is basically 
from District trade development office. The following sets of documents are examined to provide relevant data; 
registered business flies, business annual and monthly reports and entrepreneur course evaluation reports. The above 
documents generate information on educational background, average stock, number of employees, profit earned per 
month, business occupation, gender composition, business experience and monthly business sales. 

The second phase of the study collected data through questionnaires and observation. The aim of this phase 
is to provide data based on respondents as well as to test data collected in phase one of the study. The formatted 
questionnaire instruments were administered to avail data from small farm entrepreneurs and the extension trainers 
and credit providing institutions. The small farm respondents generated information concerning the following; 
enterprise identification, business training provided, respondents personal details, management skills, business 
performance. The respondents from extension trainers / creditors generated the following information; trainer/credit 
provider, respondent details, training assessment in enterprise management, rating and trend in training needs. 

Findings 

Distribution of Enterprises on the Basis of Educational Level 

Defined as follows: never gone to school / never attended formal school; lower primary / class 1 to 5; upper 
primary / class 6 to 8; secondary / form 1 to 6; post secondary / attended middle level college or university. 

Based on data analysis on the distribution of enterprises on the basis of educational levels registered 1995 
- 1999. During this period a total of 439 traders were registered. 48% of the small farm operators were of upper 
primary education being the highest category in trade registry. Followed by secondary level with 37%, post 
secondary 7%, lower primary 6% and zero educational level 2.5%. This shows a range of differing aptitude in 
knowledge application and understanding based on education factor. 

Extent Educational Levels Reveal Business Training Needs 

The educational background is assessed against the following business performance and growth indicators. 

(a) Distribution of Enterprises by Owners' Education and Monthly Turnover Ration. Stock turn over ratio is 
defined as net sales over average inventory at selling price. Distribution of enterprises by small farms’ education and 
monthly turnover ratio gives an idea on the stock management. From the data the highest number of respondents 
fall under stock turnover ratio class of 0.5 ~ 1 . The highest frequency in percentage is noted in class 0.5 — 1 under 
secondary level of education. It is however noted that 61% of the respondents have a stock turnover of one or more 
times. This indicates that 39% of the respondents can not turn the stock once. This could be due to poor stock 
control, poor pricing or stiff competition. Slow stock turnover exposes stock to the risks of loss of stock and holding 
capital in stock. A high stock turnover could be due to very low stock level hence frequent purchases being made. 
This requires proper understanding of the smallholder operators. 

(b) Distribution of Enterprises by Education and Profit. Data on distribution of stores on the basis of 
owners education and profitability shows that 50% of the no education respondents are in the profit class of 0 - 
3000. The lower primary has a peak in the class 3000 - 6000 which is 47%. The upper primary has a peak in class 
0-3000 with 25% of the respondents, while secondary has a peak in the class 15000 - 18000 with 25% of the 
respondents while post secondary has a peak in the class 15000 - 1 8000 with 33% of the respondents. The overall 
class is 3000 - 6000 with a peak of 22%. This points to the fact that with zero education the profitability from 
business are very low while at least with upper primary level the profitability improves substantially. The business 
profitability for zero and lower primary educational level is basically for survival and is within the minimum wage 
level. Educational level therefore has an influence over the profitability of the business. 

(c) Distribution of Enterprises on the Basis of Education. Distribution of business occupations on the basis 
of education level shows that over 79% of the small farm operators run retail businesses. All the zero educational 
level operate retail businesses operators in upper primary, and secondary have a wider identification in business 
occupations as compared with lower primary and post secondary. This data implies that education level has an effect 
on the business occupation one operates. The level of education has an effect on the capacity to utilize or acquire 
technical skills. 
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Attitude in Entrepreneurship 



From the data on the necessary behaviors which farm operators need to respect in enterprise management, 
the following is ranked according to the percentage of respondents; keep proper books of records (1) 100 %, consult 
trade officers (2) 95 %, concentrate on the business (3) 87 %, seeking for information (4) 77%, and direct funds to 
better projects (5) 63%. This shows that keeping proper records is most crucial behavior in respecting small farm 
business operations and shows that consultation with trade officer’s call for counseling and advisory service the time 
the small farm operator requires it. 

Knowledge and Skills Deficiencies 

The deficiencies small farm operators face in running their enterprises efficiently. Under the survey 23 
respondents were administered questionnaires. Information on deficiencies of small farm operators shows that 
credit management is the main problem area with 32%, costing and pricing 28%, recording all things 27%, 
stocktaking 22%, identifying business opportunities 20%, marketing of good 18%, profit and loss 22% and 
budgeting 13% while managing credit had 2% of the respondents. This shows that entrepreneurs require training in 
working capital management, costing and pricing, investment decisions marketing of goods, profit and loss, record 
keeping time management and business planning. 

(a) Analysis of the Stock Turnover Ratio - being net sales over average inventory at selling price. The data 
shows 48% of the respondents having stock turnover ratio between 0-1 only 8% has a turnover of one 2.5 -3. This 
shows that stock management could be poor or the businesses are under capitalized. This calls for training in 
working capital management. 

(b) Analysis of Return on Capital Employed - being the annual net profit times 100 over total capital 
employed. The data shows that over 72% of the respondents have a return on capital employed ranging between 0 - 
150% while 28% of the respondents have a return on capital employed ranging between 150% - 300%. It shows that 
the capital employed is dependent on the size of the business. This also affects the return on capital employed. 

(c) Analysis of the Cross Profit Ratio - being gross profit times 100 over net sales. This shows that 75% of 
the respondents have a cross profit ratio of between 20 - 40% while 25% have a cross profit ratio of 0 — 20%. It 
shows that business involved in provision of goods and services give higher cross profit ratios than those involved in 
retailing. 

(d) Business Occupation in relation to Sex of the Operator. The data analysis show that 70 % of the 
respondents were males while 30% were females. 84% of the males are in retail business as compared to 16% of 
females. Females have 64% in service related business as compared with 35% of males. It shows that females favor 
business occupations that add value to services and production as compared to males who prefer retail trade, general 
repair. 

Objective of Small Farm Offfarm Enterprises 

From the data on the objectives of small farm operators the ranking is as follows: To create employment for 
self and others is 93.3% of the total 60 respondents. This is followed by to provide goods/services to consumers 
with 88.3%, to earn profit / income is 83.3 and to get food is 65% pointing to the survivalist nature of the 
entrepreneurs. This shows that small farm operators basically set their business to create self employment, earn 
income and to provide goods and services to consumers. 

Small Farm Tasks 

Tasks that small farmer need to perform to achieve the goals of starting the business. Data on the tasks that 
the entrepreneur should complete to achieve the goals of starting a business through ranking shows that honesty is 
1st with 96.7 % followed by keeping accounting records with 93.3 % and employ qualified people with 86.7 %, 
open early close late with 85 %, recognize opportunities 78.7; be persistent with 66.7 % and be master of business 
58.3 % and market goods with 55 %. This shows that honesty, keeping accounting records, employing qualified 
people, opening at appropriate times, and recognizing business opportunities is important for the realization of the 
goals of starting a business. 
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Training Timing 



What is the right time for training? Information on the timings for training points at training being provided 
at least after every 6 months, followed by credit procurement and lastly before starting business. Training after 
starting business is not ranked high by the respondents. The respondents do not consider the training after starting 
business to be very important, William (1980), KIE, K-REP, KIM (1991), Nelson (1991) and Yambo (1992) look at 
counseling as an option. The data also shows that traders do not require regular training this could be because it 
distracts them from running their business. The interpretation of timing shows that training should be well timed 
with consideration of the right time for training, in order to optimize on the business returns. 

Capacities for Training 

Assessment of the existing capacities for training to support small farm enterprises was conducted. A 
questionnaire administered to extension training / credit providers gives some insights on the existing capacities, 
rating and training packages for the small farm entrepreneurs. 

(a) Capacities. The existing capacities of trainers reveal they hold either of the following qualifications; 
B.A ^cons). Diploma in Business Management and CPA II, Bachelor of Business Administration, Doctorate 
(Ph.D.) Enterpreneurship, Banking, Master of Business Administration, Bachelor of Arts (Economics) and Kenya 
Certificate of Secondary Education. 

(b) Training Package for Small Farm Operators. All the training organizations administered with 
questionnaire had training package for small farm entrepreneurs. 

(c) Types of Training Offered. Types and rating; business management 6, technical Skills 0, 
entrepreneurial Skills 1, motivational skills 1, policy procedure and credit management 1. 

(d) Adequacy of Training. Out of the seven respondents, five confirmed adequacy of the training they 
provide. Four stated that the training is not adequate. However all the seven proposed addition of counseling to 
training. Five out of seven stated that the impact of the training was good while four stated it was very good. 

(e) Rating of impact of training on enterprise management: Unknown 0, Not useful 0, Useful 6. Only six 
out of the seven organizations gave the rating. 

(0 Specific behaviors of training on trainees in enterprise management. Positive 7, Negative 0, None 0. 
All the seven considered their training to have a positive specific behavior on the creditors in small farm enterprise 
management. 

(g) Most suitable training package for entrepreneurs, (a) Business Management skills; book keeping, 
working capital management, profit & loss, business planning, investment in Business opportunities, networking and 
business linkage, basic contract law, costing and pricing, marketing and customer relations, business premises 
tribunal, weights and measures and business counseling, (b) Technical Skills; one respondent inducted that the 
small farm operators should be trained on technical skills. 

(h) Rating and trend in training needs 1995-1999. The rating by the 6 organizations which responded 
indicated that there is an increasingly trend in provision and demand for training. 

(i) Organizational Performance. The analysis of the organizational performance shows the organizations 
have not reached an optimum level in the provision of training to small farm enterprise operators management. The 
training / credit providers understand the small farm training needs and that the trainers have adequate qualifications, 
experience and are mature people. The training organizations have training package mostly in business 
management. However, 71% of the respondents consider the training they offer inadequate. They rate the training 
to be useful and have positive impact. The trainers have recommended a training package in business management. 
Despite the fact that the training organizational performance has not reached the optimum there is an increasing 
trend for training. This therefore requires proper identification of training needs so that the training is matched with 
the needs of the recipient. 

Recommendation and Conclusion 

This study recommends the need for business counseling as an integral part of extension education in small farm 
enterprise management. It reveals the importance of skills in business management and from business performance 
analysis given the views of the small farm operators and the extension training / credit providers. The following 
training areas have been identified, working capital management, identification of business opportunities and 
investment, book keeping and accountancy, business planning, marketing, human resource management, business 
networking and linkages. This can be evidenced through; giving professional advice to a client with a problem, 
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facilitating of success, sharing of knowledge, helping people explore problems so that they can decide what to do 
about them, helps reinforce the benefit of the training through guidance on practical applications, the small holder 
operator can make suitable and sustainable business decisions. The identified business management skills deficiency 
calls for training at least after every six months. The educational level plays an important role in both the success of 
a business as well as the efficient use of the training provided. The operators with no education and lower primary 
are likely not to benefit from such training because they lack the necessary capacity and orientation essential for 
business growth. Their businesses are for survival. This category generates low profit and has a low stock turnover. 
While those with upper primary level and above showing receptive to training with profitable enterprise. 

Contribution to New Knowledge in HRD 

The study helps to isolate issues that prevent holders from utilizing their potential and achieving entrepreneurial 
objectives. It identifies business counseling as a training approach to apply as part of extension education for 
appropriate smallholder knowledge and skill provision in enterprise management. Core to this study is the extend to 
which the need for knowledge, information and skills among smallholders can promote effective management of 
off-farm enterprises. Spelt out are the knowledge and skill requirements, the tasks needed for enterprise 
performance, timing of training for the entrepreneurs and the status of the trainers. There is need for training that 
matches trainers and trainees. Applying otherwise wholesome training packages without taking into account the 
trainee needs may not be effective to the recipient smallholder. Smallholders may thus operate above their potential 
and capacities for self reliance, improvement in their skills, power of intelligence and development of desired 
attitudes and values required for business performance. 
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Business-Focused Evaluation: A New Collaborative Model 



Valerie Stead 
Lancaster University 

A case study of a business-focused evaluation model is presented. This was developed as part of 
an organizational development intervention for a global semi-conductor company. The value of 
the model is assessed against three principles (concerning collaboration, joint ownership and 
integration) drawn from client and provider needs and from current thinking in evaluation 
research. Findings from the study highlight significant benefits to the Company and participants, 
and provide learning for future use. 

Keywords: Evaluation, Strategic, Integration 

Debate in the field of evaluation research and methodology indicates a number of key issues that are central to 
evaluation utilization. These include a drive towards interactive and participatory interventions that develop 
relationships with stakeholders and attend to the needs of those who are being evaluated and those who are 
evaluating (Gregory 2000; Pawson & Tilley 1997). Furthermore there is concern that interventions recognise the 
importance of how the evaluation process is managed as well as addressing the outcomes of the evaluation enquiry 
(Patton 1997, 1998; Kaghlan et al, 1999). Both of these issues link into the concept of evaluation as an intervention 
integral to organizational learning and as part of an organizational development process that aims to influence 
strategic decision-makers (Owen & Lambert 1995; Preskill & Torres 1999a, 1999b; Torres et al, 1996). 

Current literature appeared to offer little in the way of models and tools that reflect these current issues and 
that could be used to evaluate a University led intervention. This paper presents a case study outlining the design, 
development and implementation of a business-focused collaborative model of evaluation that aimed to provide a 
link between research and practice and academia and industry. The paper begins by introducing the context and 
aims of an organizational development intervention requiring evaluation. This is followed by an exploration of how 
an evaluation model has been designed around key principles identified by the organization ^etex pic) and 
Lancaster University and drawn from current evaluation research. The data analysis and findings from use of this 
working model are presented. The extent to which the model supported the key design principles is discussed and a 
number of areas for improvement are suggested. The paper is concluded by summarising the key learning from the 



Context of organisational development intervention 

Zetex pic is a global semi-conductor company with headquarters in the North West of England. In 1998 the 
company worked with Lancaster University to design a management development programme that would develop 
middle and senior managers. The programme was designed to run over one year on a part-time basis, comprising 4 
modules of 3 days and leading to a Lancaster University Certificate in Management. Participants attended four 
tutorials and produced four assignments linked to each module, and a final Personal Reflections assignment that 
aimed to reflect on their learning processes. In addition participants worked as part of a team within action learning 
groups on a business-focused project that spanned the duration of the programme. The Company initially recruited 
two cohorts of 16 managers to the programme. One cohort started the programme in October 1998 and the second 
cohort started in February 1999. 

The Company and the University were interested in evaluating the intervention in terms of learning, that is 
learning for individuals, the organization and the University. It was agreed therefore that an evaluation research 
project would run alongside the first two cohorts of students with the key aims of assessing the impact of the 
programme on the business, and assessing the impact of the programme on the individual. 

It was hoped that addressing these two organizational aims would provide the University with information 
to address academic concerns mainly, to provide an understanding of how learning impacts upon the organization 
and individuals thereby enabling further development and promotion of the programme. A further wish was that the 
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project would increase understanding of evaluation within an organizational context and would act as a test case for 
including evaluation research in other programmes. 

Methodology Rationale 

Both the Company and the University had key principles that they wanted to underpin the evaluation. The Company 
wanted an approach that would clearly link into their strategic measures, and that would enable the identification of 
specific and tangible, actual and potential benefits to the Company. The University wanted an approach that was 
integrated with the organizational development intervention and enhanced the learning for participants. It was 
therefore agreed that there were three basic principles that would underpin the approach used to evaluate the Zetex 
management development programme. The three principles were: 

Principle 1. The evaluation was to be developed collaboratively by the University team (evaluation 
researcher and programme director) and the Company (board members and senior managers) and, crucially, the 
students on each cohort. This linked into previous research by Stead at Ford Motor Company (1997) that showed 
that programme participants are the critical link between University learning processes and business outcomes. 

Principle 2. The evaluation was to be a jointly owned process between the University and the Company 
and therefore would have the capacity to deliver longer term sustainable benefits to both parties. This principle 
draws on research into participatory and collaborative inquiry and evaluation (including Gregory 2000; Cuba & 
Lincoln 1989; Heron 1996) and work by Owen and Lambert (1995) that argues that evaluators need to work 
interactively rather than at a distance and that evaluation needs to strive towards informing the ‘mental models’ of 
leaders and key decision-makers within organizations. 

Principle 3. The evaluation was to be an integral part of the learning process within the programme, and 
should link into existing strategic measures used by the Company. This principle is supported by Preskill and Torres 
(1999a, 1 999b) within their concept of evaluative enquiry, that argues the need to develop evaluation as "an integral 
on-going process that contributes to individual, team and organizational learning" (1999b, p. 93). 

There are many models and techniques to draw upon in the field of evaluation (for example, see Brown & 
Seidner, 1998; Kirkpatrick, 1998), however there appeared to be little that provided a fit with the three principles 
outlined above. Preskill and Torres (1999a, 1999b) identify seven processes at the heart of evaluative enquiry that 
aim to facilitate learning for individuals, teams and organizations and that might be used as a framework. However, 
they recognise that evaluators will need to develop new tools if ‘they wish their work to contribute significantly to 
the future success of organizations’ (Preskill & Torres, 1999b, p. 93). This management development programme 
afforded the opportunity to develop new tools that would aim to work within the concept of evaluative enquiry as 
outlined above and that would suit the specific needs identified by the University and the Company. 

Evaluation Design. 

As part of the management development programme students were asked to devise, develop and deliver a business 
project that would bring together their learning from each of the modules and that would provide a vehicle for using 
their learning in the workplace. It was decided to use the business projects as a focus for the evaluation research for 
two key reasons. First, participants were expected to work on projects of benefit to the Company and this then 
encouraged explicit linking into Company needs. Second, participants were expected to develop projects that 
reflected their learning from each of the programme modules thereby enabling a more explicit linking of application 
of learning to work. The evaluation would aim to take this critical reflection of learning and connect it to existing 
strategic measures and key business areas within Zetex pic. This qualitative approach aimed to; help students 
review their application of learning throughout the programme and project work; specify actual and potential 
achievements by students, and; provide a framework to review progress on a longer term basis. Figure I shows how 
the evaluation process was integrated within the organizational development intervention. 

With broad principles in place and a key learning focus within the programme for the evaluation, 
programme students and Zetex board members and managers were invited to a Project Launch Workshop at the 
Company. The workshop was facilitated by the programme director and evaluation researcher as the first part of the 
evaluation process with the aim of developing evaluation tools for the programme. It aimed to do this by identifying 
and agreeing what generic factors are essential for projects to be deemed successful for the Zetex business, and the 
development of Zetex managers. The workshop also aimed to identify participants’ hopes and concerns in 
undertaking the projects, and to clarify the purpose of the projects and create a pool of potential projects. 

Working in mixed groups participants, managers and board members flagged up hopes and concerns and 
identified success factors relating to projects that were then divided into two broad categories. These categories 
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were labelled hygiene factors and critical success factors. Hygiene factors were those factors that have to be in 
place before a project can proceed such as; ‘needs to have clear objectives and goals’; ‘projects should address a 
high leverage issue’. Many of these factors related directly to hopes and concerns, for example a hope that the 
project would be of value to the business, a concern that the projects would have senior management backing. 
Critical success factors were those factors deemed successful to the Company and to the participants’ development. 
Critical success factors included areas such as; ‘improved communication’, ‘add value to the business’. A pool of 
potential projects was then created by asking participants to place ideas within one of the Company’s four key 
business areas; strategic, operational, business processes and innovative. These were later circulated so that 
participants could work on them to produce project proposals. 

Figure 1. Organizational Development Intervention 



Strategic Measures (The Company) 



I 



Transfer of Knowledge (Busiijess) 



Module 1 



Evaluation process (Workshops and ujse of Tools) 

I I 

I I 

I|.earning Process (Action learning Sets) 

I I 

1 I 



Module 2 



Module 3 



Lancaster Programme 



Module 4 



Evaluation Tools 

The hopes and concerns exercise and the hygiene and critical success factors identified at the workshop were used to 
develop three tools; 

The Project Checklist was designed from the identified hygiene factors to set the context for the application 
of learning to the workplace. Project teams used the checklist to ensure that their project had the appropriate 
characteristics in place before proceeding. The project checklist included questions that project teams needed to 
consider, for example ‘will your project be easily applicable to the business?’ Once project teams had completed a 
checklist, both the Company and the University considered and agreed the proposal. The project checklist was not 
designed as an evaluation tool but was key to the evaluation process in that it enabled participants to take ownership 
of the process and to be clear about what they were aiming to achieve and its relevance to the business. 

The Critical Success Factors Map (CSF Map) as shown in Figure 2 was designed to incorporate the ten 
broad critical success factors identified at the workshop. This map formed the core of the evaluation process. It 
aimed to enable participants, the Company and the University to assess the impact of projects upon the Company 
and upon team and individual learning, by mapping actual and potential benefits. Actual benefits were those that 
came as a result of the project and could be quantifiable and tangible, for example cost savings or a new product, or 
intangible, for example improved morale. Potential benefits were those that the project team anticipated as a result 
of the project and could be quantifiable and tangible, for example increased efficiency, or could be intangible for 
example, better image. It was expected that benefits would be different for each project and that not all project teams 
would be able to identify benefits relating to all critical success factors. 

A Personal Benefits Matrix was the third tool. It aimed to assess the impact of learning upon the individual 
and included two broad ‘hopes’ identified by participants at the workshop. These hopes were ‘greater awareness and 
development’, and ‘gaining new managerial skills’. The matrix was a simple listing of students’ benefits and 
insights from the Personal Reflections assignments against the identified hopes. 
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Figure 2. Critical Success Factors Map 



CRITICAL SUCCESS FACTORS: 


ACTUAL 

BENEFITS 


POTENTIAL 

BENEFITS 


HOW WILL THE PROJECT - 

1. Add value to the business in the short to medium term? 

2. Have potential for impact on the business over the next 3-5 years? 

3. Enhance customer relationships (Intemal/Extemal)? 

4. Mobilise commitment and support from within the Company? 

5. Demonstrate communication across boundaries within Zetex? 

6. Provide opportunity for influencing/networking throughout the management 
of the Company? 

7. Demonstrate the effective management of team performance? 

8. Demonstrate the opportunity to test out the ‘Lancaster Learning’ in a work 
context? 

9. Offer the opportunity for experimentation and innovation? 

10. Create a learning environment where change is embraced? 







Data Collection and Analysis 

Participants collected data for the CSF Maps in self-managed action learning sets. As part of their project work, 
project teams met regularly in self-managed action learning sets to reflect, plan and talk through and resolve issues 
concerning their project. Each cohort of 16 participants had three project teams, thus six project teams in total. The 
CSF map was introduced to participants as a way of critically reflecting on their learning at regular intervals. Each 
project team completed a CSF Map at the start of each project and provided a final completed version for the 
evaluation research following project completion. The map was completed by specifying actual and potential 
benefits from each project for each of the identified factors. Project teams also used the CSF Map as an aid to 
developing their final project presentation to provide evidence of how their project had benefited the company. The 
Personal Benefits Matrix was completed by the research team for each cohort at the end of each programme by 
identifying and mapping key benefits and insights from the Personal Reflections assignments completed by all 
participants. The six project teams are detailed in Figure 3. 



Figure 3. Project Teams 



PI Evaluate and review group purchasing process 
P2 Understand why ‘crash’ teams succeed 
P3 Identify weaknesses in human communications 
P4 Assess methods of budgeting and re-forecasting 



P5 Address cultural and technological communications 
problems 

P6 Explore how working practices constrain productivity, 
efficiency and creativity 



Information from the Critical Success Factor Maps (CSF Maps) and the Personal Benefits Matrices of both 
cohorts of participants were analysed by linking into 8 broad areas that corresponded with existing Company 
business areas and strategic measures, as shown in Figure 4. The findings were then fed back to the Company and 
programme participants through a workshop where participants, board members and managers were asked to 
comment and add to the findings. Clearly there was an overlap between projects and areas, and the findings below 
aim to draw out some of the more salient examples. For many of the project teams their work was the opportunity to 
test out new ideas and learning and so represents the start of potential projects that could be carried forward by the 
Company. This inevitably means that there are more identified potential than actual benefits to the business. The 
findings also include data drawn from the Personal Benefits Matrices. 
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Figure 4. Areas of Analysis 



Business Areas 


Strategic Measures 


Strategic 


Financial 


Operational 


Customer 


Business Processes 


Internal 


Innovative 


Learning and Growth 



Findings 
Business Areas 

Strategic. Projects within the strategic area were defined as those projects that related to future business 
strategy or would impact on the development of new strategy. An example of actual benefits in this area is P5’s 
work of an identified strategy for improved global teamworking to include virtual teams. P6 piloted a flexible 
working system with the potential benefits of more efficient use of staff time and reduction of staff turnover. 

Operational. Projects within the operational area were those that related to Company operational activities, 
for example systems improvement. A clear actual benefit is P3’s development and implementation of a new product 
idea handling system drawing together managers from across the Company. Potential benefits include P4’s 
recommendations to improve existing budgeting procedures with the introduction of cross company sessions to 
clarify and identify budgeting information needs for new managers. 

Business Processes. These were projects that related to key business processes within the Company for 
example communications. All of the projects have benefited Business Processes in that they have been working in 
cross boundary teams and have been researching in different parts of the Company. A further benefit in this area 
would include the potential impact on customer and supplier relations and reduction in time wastage with the 
implementation of an e-commerce purchasing system piloted by PI. 

Innovative. These were projects concerned with experimentation and new ways of doing things, for 
example design of a new system. All of the projects fitted into this category. For example new ways of working 
include PTs work on the development of e-commerce, P2’s teamworking practices, P4’s work on budgeting and 
P6’s investigation of flexible working practices. New systems development includes P3’s development of a product 
idea handling system and P5’s work on an intranet employee directory. 

Strategic Measures 

Financial. This included projects impacting on the Company’s financial measures for example, increased 
cost revenue. Projects in this area were only able to anticipate financial benefits at this stage. For example, P3’s 
new product idea handling system anticipates increased orders and revenue when implemented. P6 anticipate 
reduced time to market with the implementation of more flexible working practices and cost savings with reduced 
staff turnover. 

Customer. These were projects that impacted on internal and external customers of the Company. Actual 
benefits have been evidenced by all the projects in that they have become more efficient at team-working, and they 
have learned how to work inter-departmentally. PI have improved supplier relationships by working with suppliers 
to develop a new purchasing system. P5 anticipate improved global communications including customer and 
supplier relationships with recommendations for virtual teams. 

Internal. This category included benefits to internal systems and processes. Actual benefits evidenced by 
all projects include reduced inter-departmental barriers, increased business understanding and improved internal 
networking as a result of project teams working across departments and functions to develop their projects. One 
specific example is P4 who have improved relations between finance and its customers by investigating ways of 
developing more efficient budgeting procedures. P2 and P5 focused particularly on Company team development 
and global teamworking issues, and potential benefits here include a more harmonious workforce and improved 
employee productivity. 

Learning and Growth. This included benefits to the Company and individuals, for example use of skills to 
effect change, and included data collated from the Personal Benefits Matrices. In this area there have been a range 
of actual benefits, evidenced by all teams including improved team management skills, development of new ways of 
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working, use of models and theories from the programme to promote and effect change, and increased willingness to 
learn and change. Individuals believe they have benefited greatly from working on the projects in their learning as 
part of a team, and in their personal development. Examples cited include increased confidence, a broader and more 
strategic outlook, improved management skills and the acquisition of relevant and useable theory and models. 

Discussion 

While the findings clearly show evidence of application of learning from the programme to the workplace, to what 
extent has the model enabled the initial aims of the evaluation to be met, that is to assess the impact of learning upon 
the organization and on individuals? Also to what extent has the evaluation model supported the three 
methodological principles determined during the design phase, and what suggestions does this raise for further use 
of this model? 

Initial Aims. The findings from the completed CSF Maps detail a range of actual and potential benefits 
from the participants’ project work to the organization and to groups and teams within the organization. The data 
also shows that these benefits can be related to Company strategy and have particular relevance in identified 
business areas. Data from the Personal Benefits Matrices indicate that learning has impacted upon individuals both 
personally and professionally. However the approach raised two issues concerning the aims. First, the collated data 
from the CSF maps offer more evidence of potential benefit than of specific actual benefit. This is not necessarily 
problematic or disappointing in that it does reflect the nature of the programme. For example many of the projects 
undertaken were concerned with development of innovative systems and procedures that were not implemented by 
the end of the programme and would require further detailed project planning. Therefore it is more probable that 
actual specific and tangible benefits of a project may not be evident for some time. This would then suggest that if 
the true benefits of the learning programme are to be made visible, a longer term approach to the evaluation 
intervention is negotiated with the Company. Second, examples of benefits provided by participants were not 
always detailed enough to provide clear evidence. This issue identifies a need for greater facilitation of the process 
and calls for increased integration of the evaluation process with the learning programme. This issue is addressed in 
more detail below. 

Principle J: A Collator at ively Developed Process. The development of the evaluation aimed to be 
collaborative and to include the University team, senior Company representatives and the participants. In practice 
the development occurred as the result of the University team managing a number of collaborative relationships and 
development events across the three stakeholder categories. For example the University took the lead in working 
with the Company to determine underlying principles and the key focus within the learning programme for the 
evaluation. The University facilitated a workshop for the participants and the Company to identify critical success 
factors and areas for the Personal Benefits Matrix. Having a clear understanding of which party was to manage and 
facilitate the collaborative relationships worked particularly well in the start up phase where there was commitment 
to getting the process up and running and an urgency to meet programme deadlines. However, collaboration became 
more difficult to achieve after the initial start-up when there was less urgency and focus on evaluation. In terms of 
developing the evaluation process it meant that some key steps were missed. This included taking the developed 
tools back to participants to check clarity and understanding from their point of view as product users. While this 
would have incurred extra time and cost in the start-up phase, this could have made a crucial difference to the detail 
and specificity of the data obtained. 

Principle 2: A Jointly Owned Process to Deliver Long Term Sustainable Benefits. The initial workshop 
where participants and the Company worked together to identify project ideas, hopes and concerns and critical 
success factors, set a strong foundation for joint ownership of the evaluation process. It made explicit the link 
between the programme, the organization, the project teams and the individual, and it clarified purposes and aims of 
the projects. As in the development of the evaluation process the commitment to joint ownership can be seen to 
hinge upon clear management and facilitation of the process. This is particularly evident from the way in which the 
evaluation continued. Participants worked on projects in action learning sets and were expected to use the CSF Map 
as a self-managed reflective tool. The data collation process indicated that a more interactive stance with 
participants in managing the process throughout the programme duration would have provided greater detail and 
clarity of data as well as dealing with any issues about the evaluation as they arose. This would also have avoided 
the need for the researcher to try and clarify points and add detail at the end of the process. The need to actively 
manage joint ownership is further highlighted by the issue of feedback. It had been agreed at the programme start- 
up that the University would feedback to the Company through reports and a final workshop. However while the 
nature of the feedback had been clarified, the way in which this was to be managed and the purposes of the feedback 
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had been given less attention. This meant that the Company was disengaged from this process and therefore the 
final workshop acted as an information giving session rather than as an organisational development intervention. 

Principle 3: Integral to the Learning Process and Linking into Company Strategy. The approach of 
developing the evaluation tools collaboratively and using the CSF Map as a basis for critical reflection within action 
learning sets proved very successful in linking the learning, the evaluation and the assessment into the business. The 
explicit link between learning and Company strategy was reinforced by analysing the data within the Company’s 
key business areas and strategic measures. It was felt that more learning could have been gained by the Company 
and the participants by linking the evaluation process more tightly into the learning programme and in particular into 
the assessment process. For example the CSF Map as a critical reflection tool could act explicitly as a preparation 
for the Personal Reflection assignment where participants are asked to reflect on their learning and what has been of 
most value to them. A closer link into assessment might also achieve greater joint ownership and developmental 
collaboration, in that the evaluation process becomes more visible throughout the programme duration and must be 
attended to as an ongoing process. 

The pilot of this evaluation approach also highlighted the need to develop a more collaborative and 
integrated way of assessing the impact of learning upon individuals. Although the criteria for the Personal Benefits 
Matrix were developed at the initial workshop by participants, the matrices were completed at the end of the 
programme by the University. This tool was therefore not integrated into the learning process, and did not have the 
same level of ownership as the CSF Map. Interpretation of the data from the CSF Maps showed that many of the 
benefits to the organization were also benefits to the individual for example, ‘improved team and management 
skills’. This suggests the potential for including criteria for individuals in the main evaluation tool, thus linking the 
whole process more clearly into the learning programme. In addition, the inclusion of criteria targeted at individual 
development as well as organizational development would provide a more holistic and integrated focus for critical 
reflection by attending to team concerns (the project), personal development (benefits to the individual) and the 
organization (benefits to the Company). 

A final workshop at the end of the programme presented feedback to the Company and participants. While 
the workshop explored the findings from the data there was limited exploration of how further development might 
continue. The close proximity of the workshop to final project presentations and final completion of the CSF Maps 
suggests that a workshop might be better placed some months after the end of the programme. This would enhance 
the potential for long term sustainable development by providing a focus for review of developments. It would also 
give time for some of the learning and development to be embedded within the organization, independently of the 
learning programme. 

Conclusions 

The development of a new business-focused collaborative evaluation model has brought a number of major benefits 
including: 

• Identification of specific and tangible, actual and potential benefits to the Company. 

• Identification of benefits to the individual 

• The articulation of business benefits relevant to business performance measures 

• Provision of a reflective review process and review mechanism for the organizational development 

intervention. 

• The opportunity for the University to trial a new evaluation model 

The findings have shown that for Zetex the business projects have brought key benefits to the organization 
and to individuals particularly in the development of new ideas and systems that could potentially have a long term 
and sustainable impact upon the Company. While many of the benefits specified are potential benefits there are a 
number of clear actual benefits particularly in terms of learning for the business and in terms of developing 
improved communications and internal relations within the Company. Of particular interest is the number of 
innovations developed and piloted through the business projects with the aim of improving Company productivity, 
technology and market share. The findings also show that individuals gained enormously from being involved in the 
development of business projects. This increased confidence and self-esteem has had a beneficial effect in the 
workplace in terms of feeling more able to make decisions, to suggest improvements and to carry them through. 

The evaluation process has aimed to be a collaborative and jointly owned process between the university, 
the Company and the participants. It has also aimed to be integral to the learning process and to link into Company 
strategy. Overall the pilot has been successful in achieving its aims and has identified some key learning points for 
the research team and highlighted a number of areas for improvement to the overall process. 
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The pilot has provided the opportunity to trial a new evaluation model linked into current evaluation 
research thinking. This has reinforced the need to attend to process both in how process is managed with other 
stakeholders in the initial start-up and in the ongoing development phases. Attention to process management has 
also been shown to be crucial in how participants work with the evaluation within the organizational intervention, 
for example having an opportunity to check understanding, and using the evaluation process as a learning tool. 
Working within a joint owned process has also raised the issue of commitment. For example, the pilot has indicated 
the need for stakeholders to think beyond the evaluation to how tools, frameworks and processes can help continued 
development within the organization. There has also been a clear indication from the pilot that there is potential in 
adding to the learning from the evaluation and adding to the value of the programme by linking this approach more 
tightly into the assessment process. 

In conclusion our learning from this evaluation research has developed the basis for a strategic approach to 
evaluation as part of a learning and organizational development process. The evaluation model is now being refined 
and developed across a range of work-based and academic qualification programmes within Lancaster University. 
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Challenges in Measuring and Evaluating Diversity Efforts in HRD 



T. Marshall Egan 
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This article examines the challenges faced by practitioners and scholars interested in evaluation 
and measurement of diversity efforts in organizational contexts. An organizational systems model 
illustrating the diversity related interchanges between an organization and the macro environment 
are considered. The challenges to measurement and evaluation due to the latency of the diversity, 
performance and performance improvement constructs are examined. 

Keywords: Diversity, Evaluation, Performance Improvement 

Practitioners and researchers are currently in pursuit of greater understanding regarding the impact of diversity and 
diversity management practices on organizational performance. In the interest of performance, global or 
multinational corporations frequently develop strategic approaches to enhance the likelihood of success in 
expanding market share among an increasingly global community of customers (Albert, 1994; Dass & Parker, 
1996). Recruitment, hiring and retention of a diverse workforce are essential for organizational success in an 
increasing number of localities and industries. Since the late 1980s, many organizations have engaged in a response 
to workplace and marketplace diversity through initiatives focused on the management of customer, supplier and 
employee diversity (Kossek & Lobel, 1996). Regardless of the approach to diversity, organizations are engaged in 
developing more specific knowledge regarding the measurement of outcomes of diversity efforts and how they 
might be improved (Wheeler, 1996). 

Purpose of the Paper 

This paper explores the question: What are the current challenges to measuring the effectiveness of organizational 
diversity efforts? The exploration of this question will be accomplished through a review of the literature, 
description of current organizational practices associated with diversity efforts and an analysis of related approaches 
to measurement and evaluation. The limitations to measuring and evaluating the effectiveness of diversity efforts 
will be explored at the organizational systems level. 

Defining Diversity 

Many organizations have adopted definitions of diversity that are broad in scope (Wheeler, 1996). “One of the most 
powerful concepts underlying successful diversity initiatives is a broad, inclusive definition of diversity” (Hayles & 
Russell, 1997, p.l 1). Broad definitions extend beyond a categorization system centering on race, gender, age and 
disability to include a variety of differences. Thomas (1999) emphasizes that definitions of diversity should not 
focus exclusively on minorities and women in the workforce, but instead should include “any collective mixture 
characterized by differences and similarities” (p. 50). Although many organizations maintain a broad definition of 
diversity, communication and practice often focus more narrowly on categories associated with equal opportunity 
employment (Thomas, 1999). For the purposes of this paper, the definition of diversity will be the mixtures of 
individual characteristics in the context of an interdependent human system whereby associated similarities and 
differences or perceptions of similarities and differences influence norms, behaviors, interaction, and performance. 
The term diversity effort will be used to describe an activity designed to support the definition and diversity related 
goals for an organization. 

Conceptualizing the Forces Influencing Organizational Diversity Efforts 

Johnson (1995) found that over seventy percent of Fortune 500 companies either had or were planning diversity 
efforts. The following four forces can be identified as strong influences on United States and global diversity 
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efforts: diversity action, diversity inquiry, diversity knowledge, and diversity infusion. The concept of diversity 
forces relates to a collection of events that influence organizational decisions to engage in diversity efforts. These 
diversity forces are defined at the macro systems level and work across systems, influencing and being influenced 
by the components of the system such as organizations, groups and even individuals (Jacobs, 1989; see discussion 
on systems theory below). 

Diversity action is any activity, program, law, policy or process that influences collective perception and 
behavior regarding similarities and differences within human systems. Diversity inquiry is the act of examining the 
existing collective process as it pertains to the normative, behavioral and perceptual influences of similarities and 
differences in a human system. Diversity knowledge is the insight gained individually and collectively that 
influences reasoning, interpretation, planning and action within a human system. Diversity infusion is the ongoing 
introduction and integration of persons into a group, organization or community or the development of awareness of 
underlying characteristics resulting in increased explicit heterogeneity of a human system. 

Diversity action is associated with diversity related legal parameters for organizations, protests against 
organizations perceived to be discriminatory, and celebrations of ethnic or cultural identities. Diversity action is 
evidenced in the workplace and the marketplace (Cox, 1994). As diversity awareness in organizations has 
increased, so has diversity inquiry that involves organizations becoming active in researching specific market trends, 
consumer demographics, and potential for increasing new demographic market segments. Organizations also 
responded to customer and worker inquiry regarding organizational behavior and commitment to diversity (Wheeler, 
1996). Political campaigns, organizational advertising and public service announcements began to focus on 
diversity. Increases in related activity have produced greater interest in diversity knowledge. Public and corporate 
interest in diversity sparked the publication of numerous books, articles and videos along with extensive media 
coverage of diversity issues. The increased focus on diversity led to training that increased diversity knowledge 
(Hanover & Cellar, 1998). Changes in organizational openness to diversity issues and the introduction of larger 
numbers of persons with visible differences contributed to diversity infusion. This influx of new people increased 
heterogeneity of organizational membership (Judy, 1997). The culmination of macro environmental forces 
associated with diversity led to multifaceted organizational approaches to addressing diversity issues among many 
organizations (Kormanik, Krieger & Tilghman, 2000). 

Diversity efforts have been tied to organizational goals (Wheeler, 1994). “Practitioners agree that multiple 
elements are essential in the process of creating an environment that values diversity. No single approach is 
sufficient” (Wheeler, 1994, p. 21). The reason for a broad approach to diversity efforts is that the issues or drivers 
are too complex. In a study of diversity efforts in several large corporations, Wheeler (1996) identified six key areas 
of emphasis including: communication; education & training; employee involvement; career development and 
planning; performance and accountability; and culture change. These areas suggest the possibility for a systematic 
approach. “A systematic approach that addresses issues at multiple levels through planned stages with continuous 
reinforcement overtime is characteristic of diversity pioneers” (Wheeler, 1994, p. 21). 

An Organizational Systems Model for Diversity 

The first part of the systems model below (depicted in the cube formation at the center) represents seven key 
organizational practices in alignment. The cube is situated within the permeable internal environment of the 
organization. Because the model is located in a naturalistic environment, alignment and realignment are ongoing. 
The lines representing the edges of the cube do not dominate the relationships between each element. Rather there 
is interchange between the permeable walls of the cube, so that each practice area influences the other, though not 
necessarily in a proportionate manner. Although every organization is unique, these core practice areas must be 
addressed to one degree or another in all organizations. Ineffective management of one or more areas, due to lack of 
systematic maintenance results in the organization becoming misaligned. 

The core practices making up the cube are modified from the Malcolm Baldridge Award (George, 1996). 
To align the practices, measures must be derived and monitored. Alignment is important to the success of 
organizations, but often difficult to accomplish. Although alignment may not guarantee organizational success, 
misalignment is problematic (Semler, 1997). The model represents an organization striving to maintain alignment 
between key areas of practice. The organization system uses evaluation, measurement, planning and readjustment in 
an ongoing attempt to stabilize itself. 

The second part of the model represents the macro system environments and includes the internal 
organizational, industry and macro environment. Permeable lines signify the interchange between these 
environments. The macro environmental features that may influence the industry and internal organizational 
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Figure 1. A Diversity Focused Organizational Systems Model 



Kconomic 




environment are global, demographic, technological, political, economic, social, and legal factors. Industry 
environmental features include: competitors, product and service substitutes, suppliers, and customers. The double 
tipped arrows depict diversity forces. The arrows represent the dynamic interchange occurring across all levels of 
the system. Organizational responsiveness to the diversity forces as they permeate the organization impacts 
personnel, policies, new issue related insights, and questions. The organization in turn initiates interchanges that 
influence the industry and macro environment. 

The three sections described and depicted above represent an open system where permeable boundaries 
allow for a constant state of interaction. These dynamic interchanges demand that the organization make ongoing 
adjustments to diversity forces. The interchange may be viewed as providing challenges and opportunities in the 
organization. The organizational reaction to the environment is often dependent on the information gathered 
through evaluation or measurement. Additional information may provide an opportunity for the organization to 
define interchanges with diversity forces as an opportunity. 

Awareness of existing diversity issues within the organization obtained through diversity inquiry and 
knowledge combined with the interactions within an increasingly heterogeneous macro system present new 
challenges. The organization is positioned to maintain short and long term goals in the face of conflicting 
stakeholder interests. For example, managers may be expected to meet the demands of shareholders for return on 
investment, customers for quality and affordability, employees for fair wages, and communities for pollution control 
(Dass & Parker, 1996). The fact that organizational stakeholders at all levels of the system rarely agree is combined 
with the uncertainty that “investment in diversity” is a profitable strategy. At the same time, the organization has an 
opportunity to influence the industry and the macro environment with regard to diversity actions such as policies 
regarding supplier relationships or the creation of products addressing a certain segment of the available market. 

Organizational Systems Theory in Measurement and Evaluation 

Argyris and Schon (1996) have emphasized the importance of systemic approaches to organizational planning and 
evaluation through their work on organizational learning. There is limited available scholarly literature regarding 
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diversity at the organizational systems level (Hayles & Russell, 1997). Dass and Parker (1996) developed an open 
systems model that represents the permeable borders between organizations and their macro environments. The 
model represents the open systems environment interchange that occurs between the industry environment, and the 
organizational environment. According to Dass and Parker, organizational members must be aware of both internal 
and external concerns associated with diversity in order to be in the best position to add value and to achieve a long- 
term strategic advantage for the firm. Strategic management is important for ongoing and flexible processes in 
which organizational leaders define organizational mission, set goals and priorities at various levels, and develop 
and implement means to achieve them. 

From the practitioner literature, the Griggs (1994) model focuses on “systemic human resources and 
management development challenges”. This model is segmented into six parts including: total quality, change, 
leadership, productivity, relationship, employee participation, and new work formats. The model provides an 
opportunity for the understanding of various linkages and ways in which a business environment is impacted by 
diversity. Hubbard’s (1997) “Diversity 9-S Framework” (strategy, structure, systems, skills, staff diversity, style, 
shared values, shared vision and standards) utilizes an approach to measuring organizational diversity efforts. 
Although systems theory is not the explicit focus, the stated intention of the 9-S Framework is to be comprehensive 
at the organization level. Most of Hubbard’s measurement formulae are modified from the existing human 
resources measurement literature, particularly the work of Fitz-Enz (1984). Although Hubbard’s model and 
measurement concepts appear to have been widespread in business and industry, the underlying theory of and results 
from the approach are unknown. Hubbard’s model, however, is one of the most thorough to date because of its 
attempt to conceptualize a broad array of human resource issues associated with diversity. 

The Role of Diversity Measurement and Evaluation in Organizational Contexts 

The most common utilizations of organizational measurement and evaluation are prediction, explanation, 
management of programs, and the maintenance and sustainability of current and future performance (Holton, 1997). 
Measurement and evaluation can be used to influence planning. Planning is often a product and mediator between 
diversity evaluation and measurement, and provides the drivers for performance improvement that are linked to 
organization success (Kaplan & Norton, 1996). From this perspective, measurement alignment is important to the 
success of organizational functioning ^emler, 1997). Alignment denotes linkages among an organization’s 
strategic plan, organizational management style, reward and recognition systems, and external customer concerns. 
Rummler and Brache (1995) suggest that data collected related to the organizational context represent the 
performance situation and form the basis for making decisions. Managers may use data to monitor core 
organizational systems and processes; employees may use the same data to improve individual job performance. 
Data can also be used as a communication medium among managers and employees for identifying current and 
expected performance. 

Measuring organizational performance to include diversity related performance offers the opportunity for 
higher levels of sophistication in approaching measurement in the organizational context. Evaluation of 
organizational practices from a systems perspective presents more complexity. When measurement and evaluation 
are well defined they can be better understood and implemented. Decisions regarding the use of attributes or 
indicators are requisite for individual, team and organizational success in improving performance (Sleezer et al. 
1998a). There are necessary linkages between evaluation, measurement and planning of diversity efforts (Thomas, 

1999) . 

Evaluation and Measurement of Diversity Efforts 

Although several organizations have been involved in diversity interventions for more than a decade, there is limited 
evidence of evaluation associated with diversity initiatives in the HRD literature. Even fewer articles or reports 
exist regarding the scope and quality of those measures (Wentling, 2000; Wheeler, 1994). Comer and Soliman 
(1996) emphasize the importance of evaluating the results of diversity efforts. “Evidence of diversity’s impact on 
the bottom line has not been systematically measured and documented for easy retrieval and use” (Robinson & 
Dechant, 1997, p. 21). To date, most academic research has been primarily concerned with the impact of race and 
gender in dyadic relationships and groups (Kossek & Lobel, 1996; Richard, 2000). Applied research has focused on 
the management of workforce diversity including recruitment, hiring, selection, and utilization of diversity (Richard, 

2000 ) . 

Recent findings indicate that organizations have begun to acknowledge that measurement of diversity 
efforts is of importance and have begun to engage in evaluation practices associated with diversity initiatives 
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(Wentling, 2000). In Egan’s (2000) study of US corporations, approximately seventy-five percent of respondent 
organizations reported some use of evaluation measures associated with diversity efforts. There are, however, 
indications that evaluation of diversity efforts is new to organizations and that evaluation and measurement 
effectiveness are of significant concern (Wentling, 2000). Wentling indicates the possibility of an often-narrow use 
of instrumentation by organizations. 

Most of the available reporting of corporate diversity measurement appears to be broad in scope and does 
not provide adequate information to determine the specificity and frequency of reported diversity measurement 
approaches. One example is that of Marriott International, Inc. (Tate, 1999), Marriott has taken a business 
objectives approach. The company has identified measurement as being critical because of a belief that tangible 
evidence “gets noticed”, indicates progress, can be supported in the budget, and is vital to ongoing refinement and 
adjustment (Tate, 1999). Marriott’s existing measurements include: attitude surveys; retention records of minorities 
and females; EEO complaints filed; documenting representation by minorities and women in senior, middle and 
entry level management ranks; tracking legal fees and settlement costs for discrimination cases; minority and female 
vendor and supplier relationships; recruiting and ability to attract diverse candidates; minority and female 
franchisees; external recognition; and benchmarking. Additionally, rewards to managers are given for: 
“programmatic and numeric diversity objectives”; examination of corporate policies and practices to ensure that 
policies are supportive of diversity endeavors; and inclusion of a diversity component in performance evaluations. 
This example demonstrates the following: 1) the measurements are context specific; 2) there are a range of 

approaches to diversity performance measurement; and 3) the measurements link two or more performance domains 
including mission subsystem, process, team, and individual levels (Holton, 1997). 

Key Factors to Evaluation and Measurement Effectiveness 

Definitions vary for the terms measurement and evaluation. Measurement is most often described as an approach or 
process for gathering information or data collection. Evaluation has been defined as the determination of 
congruence between performance and objectives ^ehrens & Lehmann, 1991) and as a process for providing 
information about activities and organizational elements to influence decision-making. Evaluation has also been 
defined as a process for determining the worth or quality of something (Worthen & Sanders, 1 987). 

Kettner, Moroney and Martin (1999) define measurement as a system of rules usually focusing on the 
process of numerical assignment to phenomena. Nunnally and Bernstein (1994) identify measurement as consisting 
of rules for assigning symbols to objects. The purpose of these assignments are “scaling” and “classification”. 
Scaling is referred to as the numerical representation of attributes while “classification” determines whether objects 
fall into the same or different categories with respect to a particular attribute. 

Nunnally and Bernstein’s (1994) use of the words attributes, rules and scaling are important considerations 
(Sleezer, Hough & Gradous, 1998a). Attribute refers to our inability to measure an object or characteristics of an 
object directly. Although we are not able to measure attitude directly, we can measure indicators such as the 
observed numerical values on a survey instrument. Similarly we may be able to directly measure the size, color and 
price of an appliance in a store, but we must infer quality by measuring observable indicators (Sleezer et al. 1998a). 
Most of the elements associated with organizational performance and diversity must utilize proxies. 

The term rules, as emphasized by Nunnally and Berstein (1994) are also important ^leezer et al. 1998a). 
The assignment of numbers must be unambiguous and clearly stated. Rules for the measurement of attributes are 
not always explicit. The rules may be a product of deduction, be based on previous experiences, flow from 
intuitions or common sense, or be a product of estimation (Nunnally & Bernstein, 1994). “Because the rules for 
measuring performance are not intuitively obvious, performance has been assessed in multiple and sometimes 
conflicting ways” (Sleezer et al. 1998a, p. 64). Some of the examples of process quality measures that do have 
structured systems of rules for assigning numerical values include ISO 9000 guidelines and the Baldridge Award 
Criteria (George, 1996; Schlickman, 1998). The challenges presented by such systems are that the rules for 
assigning numerical values are negotiated and change over time (George, 1996). From the examples above, 
evaluators creating measurement systems must determine the performance attributes to measure, the indicators that 
will be used to measure them and the rules for assigning numbers. In order to ensure credibility of the evaluation 
effort, measures must be reliable, valid and practical. Reliability is associated with consistency and accuracy, and 
refers to the ability of an instrument to measure the same thing when used several times. Validity ensures the 
instrument measures what it is intended to measure. Practicality refers to the ability to administer the instrument in 
a reasonable amount of time with realistic administrative requirements. 

Even the best measures are not absolutely consistent (Borg & Gall, 1989). Some organizations 
participating in diversity related performance improvement efforts survey customers based on specific demographic 
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categories. One approach to such research is the use of a customer survey to determine product performance. There 
are several potential sources of measurement error in this case including: the customer, situational factors, survey 
instrument, and the measurer. Some customers may provide incorrect demographic data, or overstate the use of the 
product. The environment where the survey is being distributed and collected may not be conducive to produce 
open disclosure. Accuracy may be compromised if the measurement instrument contains ambiguous questions or 
the measurer holds unexamined biases or utilizes improper statistical techniques. Measurement error is particularly 
problematic when attempting to measure abstract and theoretical concepts (Fowler, 1993). 

The process for implementing any quantitative' measurement system is a product of negotiation and 
renegotiation over time. Performance improvement measures evolve, as do those individuals who manage and 
interpret the data (Sleezer & Gradous, 1998b). No measure of performance or of organizational systems is perfectly 
consistent. Measurement errors, situational factors, measurement instruments, subjects of measurement, and 
measurers can all contribute to inconsistencies. Although there are ways to control for some errors, inconsistency in 
measurement will continue to be a challenge for researchers (Borg & Gall, 1989). 

According to Borg (1987) there are four main uses of scales in social science research. Interval and ratio 
scales are often called metric while nominal and ordinal scales are commonly referred to as nonmetric. Scaling 
refers to the numerical statement that can be made about how much of an attribute is present (Nunnally & Bernstein, 
1994). Measuring diversity performance requires the use of the appropriate scale according to the situation and the 
data. Nominal scales involve typical identifiers that are static along a dimension such as gender or “racial category”. 
Ordinal scales are simple scales that order people, objects or events along some continuum, like organizational rank, 
but do not report anything about the differences between units on a scale. Differences along interval scales have the 
same meaning anywhere along the scale such as on a thermostat. Ratio scales have a true zero point, such as salary 
(Howell, 1992). According to Nunnally and Bernstein (1994) metric measuring scales make available more 
information than categorical scales and should be used when possible. The addition of metric scales when 
measuring and evaluating diversity can produce more meaningful and complex indicators of related performance. 

The definitions of evaluation and measurement explored above will assist in focusing relevant questions 
regarding the measurement and evaluation of diversity efforts in organizations. A first question for an evaluator of 
diversity is: How is diversity defined? How diversity and organizational diversity efforts are defined will determine 
the selection of attributes. These attributes may be either directly measurable or abstract, latent constructs. Latent 
constructs cannot be measured but observable indicators that infer the construct can provide measurements. When 
examining the evaluation of diversity there are several constructs at play, none of which can be measured directly. 
The three major constructs associated with measuring diversity efforts are diversity, performance and performance 
improvement. Sleezer et al. (1998b) defines performance in the following way: "Performance consists of processes, 
accomplishments, and capacities that affect current and future individual, team, and growth in capacities” (p.l23- 
124). Attributes and their respective indicators are used to infer performance. The concept of customer service to a 
particular demographic group could be used to infer performance results for a business unit. Asian-American 
customers could be asked questions to elicit information regarding various indicators. By combining the answers, 
we could develop a view of the customer service construct and ultimately performance. We could also gain a 
perspective regarding the customer service to minorities construct and ultimately insight on diversity. In the case of 
both performance improvement and performance improvement of diversity, we must enumerate processes, 
capacities and accomplishments that are changing or have changed. Once quantified, we can use measurement to 
gauge attributes of a performance prior to and following an improvement activity or to test attributes of the activity 
itself (Sleezer et al. 1998a). 

Holton (1997) proposed four performance domains for organization systems. The author identified typical 
metrics that he classified as performance outcome measures or performance driver measures. Sleezer et al (1998a) 
indicated whether the performance outcomes or drivers were constructs, indicators or not defined. A chart 
specifying diversity-related measures as constructs or indicators is featured below (Chart 1). Most of the 
performance drivers and outcomes presented are latent constructs that provide special measurement challenges to 
those interested in performance improvement. The constructs are difficult to identify by observations, objects or 
specific events. The challenges are even greater when constructs associated with diversity are included. 

Evaluation and measurement of diversity efforts are challenging. In order to measure meaningful outcomes 
associated with diversity, diversity must be linked to performance and performance improvement. Using 
measurement to evaluate the effectiveness of diversity is complicated for the following reasons: diversity and 
performance are latent constructs, as is the concept of performance improvement related to diversity; these latent 
constructs have multiple attributes that may be confounded by other attributes such that they are not distinguishable; 
indicators may connect to multiple attributes; multiple indicators exist for each of the attributes; and, as supported 
by the definition of diversity, each situation being measured involves a unique combination of attributes. The 
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Chart 1. Typical Performance Outcomes and Drivers for Organizational Systems 
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difficulties involved with these issues along with the relative infancy of diversity efforts in organizations suggest the 
need for research that builds performance measurement theory and links it to practice. 

Conclusion 

The challenges discussed above have implications for the measurement of the effectiveness of diversity efforts. 
Further research is needed to target individual interactions among and between practice areas within organizational 
systems and between the internal organizational environment and the industry and macro environments. Within 
each of the interactions examined, diversity related performance situations and their contexts should be studied and 
the complexities of measuring diversity performance must be acknowledged. Understanding the elements of 
diversity performance within a particular interaction will provide a baseline for utilizing evaluation to improve 
diversity-related organizational performance. 
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